System and method for providing deep learning-based virtual reality 3d embryo model

ABSTRACT

The present invention discloses a virtual reality embryo image providing system. More specifically, the present invention relates to a deep learning-based embryo image providing system which extracts facial features of an embryo from an ultrasound image on the basis of a deep learning technique, generates a 3D model corresponding to the ultrasound image reflecting the facial features, and provides a virtual reality image using the 3D model. According to an embodiment of the present invention, the figure of an embryo can be displayed three-dimensionally through an HMD or the like by setting a plurality of codewords reflecting the features of each body part for a 2D embryo image, and performing a learning procedure on the basis of the codewords according to a deep learning model to provide a 3D model generated by combining the body components that are most similar to the actual face of the embryo. Therefore, a differentiated and realistic embryo imaging service can be provided to a pregnant person.

TECHNICAL FIELD

The present invention relates to a system for providing a virtualreality 3D fetal model, and in particular, a system for providing a 3Dfetal model based on deep learning that extracts facial features of afetus based on deep learning technology on an ultrasound image,generates a 3D model reflecting the facial features, and providesvirtual reality content using the same.

BACKGROUND ART

Currently, fetal ultrasound images are typically 2D flat images, and areused only for viewing purposes, e.g., by being directly viewed throughthe monitor by the pregnant woman or being stored in video format. Mostof the conventional maternal apps and related services only provideinformation on pregnancy and childcare, or simply play video services.It is believed that this is due to the fact that technical developmentor service enhancement in related fields has not yet been made, andkiller applications are insufficient.

Technologies for effectively visualizing fetal ultrasound images,especially 3D fetal faces, may be an important factor for enhancingrelated medical services. However, unlike conventional facialrecognition targeting general facial images, image processing targetingthe fetal face of an ultrasound image is extremely hard. This isbecause, by the nature of the ultrasound fetal face image, such featuresas eyes, nose, and mouth are unclear, and there are various variablesdepending on the location of the tissue in the uterus of the pregnantwoman or the fetal face.

Accordingly, it is not an easy task to read the 2D image and apply depthinformation then convert it into a 3D stereoscopic image, and adifficulty exists in implementing such a service as to provide 3D fetalultrasound images through a virtual reality display means, e.g.,head-mounted display (HMD).

Prior art documents in the technical field to which the presentinvention pertains include Korean Patent Application Publication No.10-2013-0112127.

DETAILED DESCRIPTION OF THE INVENTION Technical Problems

The present invention was conceived to solve the foregoing problems, andthe present invention aims to implement a fetal image similar to theactual fetal face on a virtual reality device by extracting featurescorresponding to the actual fetal face and generating a 3D fetal modelreflecting the extracted features.

Means to Address Problems

To achieve the foregoing objects, according to an embodiment of thepresent invention, a system for providing a virtual reality 3D fetalmodel based on deep learning may comprise an image providing servergenerating a 3D model by analyzing a fetal image provided from a mobileterminal. The image providing server may include an image receiving unitreceiving a fetal image and ROI information from the mobile terminal, animage processor inputting a fetal image corresponding to the ROIinformation to a deep learning model and extracting per-facial portionfeatures, and a 3D model generator generating a 3D fetal modelreflecting the per-facial portion features by selecting and synthesizingpart models according to the per-facial portion features.

The image processor may include a preprocessor designating one or moreframes from the input fetal image and removing noise; and an ROImatching unit determining a facial area, as a target for modeling, bymatching the ROI information onto the frame.

The image receiving unit may receive a codeword for a similar modelcomposed of a plurality of components selected from the mobile terminalalong with reception of the fetal image and the ROI information. Theimage processor may include a deep learning model for performing amachine learning process using the codeword corresponding to theplurality of components, as training data, or extracting a facialfeature codeword for an input image after training.

The facial area may include one or more of an eye, a nose, a mouth, anda face. The codeword may be a code resultant from a shape of the facialarea in a binary format.

The image processor may include a component selector exchanging one ormore components for each facial area included in an existing 3D model,corresponding to the codeword.

The image providing server may include a member management unitdetermining whether a user is registered as a member according to logininformation input from the mobile terminal and performing a loginprocess. The image processor may store the fetal image in storageallocated to a user identified by the member management unit and extractthe stored 3D fetal image.

The image processor may receive growth information including a pregnancyperiod and growth parameter for a pregnant woman from the mobileterminal. The 3D model generator may store the generated 3D model in adatabase according to the growth parameter per pregnancy period.

The mobile terminal may include a communication unit connected to aninformation communication network to communicate with an externalsystem, a storage device storing the fetal image transmitted from anultrasound diagnosis device through the communication unit, an ROIsetting unit receiving a selection of a facial area in the fetal imageaccording to a user input and setting the facial area as the ROIinformation, a growth information input unit receiving growthinformation including a pregnancy period and growth parameter for apregnant woman, a terminal library DB storing a library in whichcomponents corresponding to a plurality of face shapes are defined, anda feedback unit receiving a selection of one or more components forfacial portion of the fetus from a user and transmitting a codeword of asimilar model to the image providing server.

To achieve the foregoing objects, according to another embodiment of thepresent invention, a method for providing a model by an image providingserver of a deep learning-based virtual reality 3D fetal model providingsystem may comprise receiving a fetal image and ROI information from amobile terminal, extracting per-facial portion features by inputting afetal image corresponding to the ROI information to a deep learningmodel, generating a 3D fetal image reflecting the per-facial portionfeatures through 3D modeling, and providing the 3D fetal image to themobile terminal.

The method may further comprise, after providing the 3D fetal image tothe mobile terminal, receiving a codeword for a similar model includingone or more components per facial portion of the fetus selected by auser from the mobile terminal and performing a machine learning processon the deep learning model using the codeword as training data.

Advantageous Effects

According to an embodiment of the present invention, a plurality ofcodewords in which features are reflected for each body part of a 3Dfetal image are set and, based thereupon, a learning process isperformed according to a deep learning model to thereby provide a 3Dmodel generated with a combination of body components most similar tothe actual fetal face. Therefore, the appearance of the fetus may bestereoscopically displayed through, e.g., an HMD, thereby providing adifferentiated and realistic fetal image service to pregnant women.

Further, the fetal imaging service according to an embodiment of thepresent invention may be used in health care applications for fetusesand pregnant women, such as prenatal care and depression treatment forpregnant women.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a view schematically illustrating an overall configuration ofa deep learning-based virtual reality 3D fetal model providing systemaccording to an embodiment of the present invention.

FIG. 2 is a view illustrating a structure of a mobile terminal of a deeplearning-based virtual reality 3D fetal model providing system accordingto an embodiment of the present invention.

FIG. 3 is a view illustrating a structure of an image providing serverof a deep learning-based virtual reality 3D fetal model providing systemaccording to an embodiment of the present invention, and

FIG. 4 is a view illustrating a process for increasing the matchingaccuracy of a 3D model by reflecting user feedback in a deeplearning-based virtual reality 3D fetal model providing system accordingto an embodiment of the present invention.

FIG. 5 is a view illustrating a structure of an image processor of animage providing server according to an embodiment of the presentinvention, and

FIG. 6 is a view illustrating a 3D modeled image of an image providingserver according to an embodiment of the present invention.

FIG. 7 is a view illustrating a method of providing a virtual reality 3Dfetal model based on deep learning according to an embodiment of thepresent invention.

FIG. 8 is a view schematically illustrating a deep learning process (a)in a deep learning-based virtual reality 3D fetal model providing systemaccording to an embodiment of the present invention and a codewordextraction process (b) using the learned deep learning model.

MODE TO PRACTICE THE INVENTION

Throughout the specification, when an element “includes” anotherelement, the element may further include the other element, ratherexcluding the other element, unless particularly stated otherwise.Further, the term “unit,” “device,” or “system” as used herein denote aunit processing at least one function or operation and be implemented inhardware, software, or a combination thereof.

Although some exemplary embodiments of the disclosure are describedherein, the technical spirit or scope of the disclosure are not limitedthereto. Prior to going into the detailed description of the disclosure,it might be effective to define particular words and phrases as usedherein. As used herein, the terms “include” and “comprise” and theirderivatives may mean doing so without any limitations. As used herein,the term “or” may mean “and/or.” As used herein, the phrase “associatedwith” and “associated therewith” and their derivatives may mean“include,” “be included within,” “interconnect with,” “contain,” “becontained within,” “connect to or with,” “couple to or with,” “becommunicable with,” “cooperate with,” “interleave,” “juxtapose,” “beproximate to, “be bound to or with, “have, or “have a property of.” Asused herein, the term “controller” may mean any device, system, or partthereof controlling at least one operation. As used herein, the term“device” may be implemented in hardware, firmware, software, or somecombinations of at least two thereof.

Various schemes or methods described herein may be implemented inhardware, software, or a combination thereof. As used herein, the term“unit,” “device,” or “system” may also be equivalent to acomputer-related entity, a hardware component, a software component, ora combination thereof. Each function executed in the system of thepresent invention may be configured as a module unit and may be recordedin one physical memory or be distributed and recorded between two ormore memories and recording media.

Hereinafter, a deep learning-based virtual reality 3D fetal modelproviding system and method according to an embodiment of the presentinvention is described with reference to the drawings.

FIG. 1 is a view schematically illustrating an overall configuration ofa deep learning-based virtual reality 3D fetal model providing systemaccording to an embodiment of the present invention.

Referring to FIG. 1, a deep learning-based virtual reality 3D fetalmodel providing system according to an embodiment of the presentinvention may include a user's mobile terminal 100, an image providingserver 200 that interworks with the mobile terminal 100 to convert a 3Dfetal image into a 3D model using a trained deep learning model, anultrasound diagnosis device 300 that photographs a pregnant woman andprovides the 2D fetal image to the mobile terminal 100, and a VR device400 implemented as, e.g., an HMD to display the 3D model as a virtualreality image.

The mobile terminal 100 is a terminal that is possessed by the user,such as a pregnant woman or her guardian and is connected to the imageproviding server 200 through an information communication network totransmit the fetal image stored in the terminal to the image providingserver 200 to thereby request a 3D model for the fetal image.

As the user desires to view a more realistic image of the fetus with theVR device 400, the user may transmit her fetal image stored in hermobile terminal 100 to the image providing server 200 through theinformation communication network.

To that end, the mobile terminal 100 previously needs to receive andstore a 2D original fetal image from the ultrasound diagnosis device 300through wired or wireless communication.

Further, upon transmitting the fetal image, the mobile terminal 100further receives, and transmits together with the fetal image, growthinformation including what week of pregnancy or growth parameters of thepregnant woman, thereby receiving the 3D model updated according to thedegree of growth of the fetus from the image providing server 200.

As the mobile terminal 100, a smartphone or tablet PC may be used whichincludes a communication module, microprocessor, memory and storage thatmay store and execute an application program capable of interworkingwith the image providing server 200.

The image providing server 200 may interwork with a plurality of mobileterminals 100 through the information communication network and storefetal images transmitted from each mobile terminal 100 and may use thefetal images as data for machine learning. Further, the image providingserver 200 may separate parts of the fetus's face into, e.g., eyes,nose, and mouth and designating features of the face shape, extract themost similar components through comparison with a model libraryaccumulated in the database, determine a 3D model corresponding to thefetal face through a combination of the components, and provide thedetermined 3D model, as a 3D fetal image, to the mobile terminal 100.

As the image providing server 200, a server device equipped with ahigh-performance microprocessor and high-capacity memory and storage maybe used that may process, without delay, the tasks of performing machinelearning on the fetal images transmitted from the multiple mobileterminal 100, generating and providing a 3D model, and updating the 3Dmodel according to the growth of the fetus.

The ultrasound diagnosis device 300 may include a predetermined probe toobtain acquire an ultrasound image, which is an original image of a 3Dmodel, for the diagnosis target.

Further, the ultrasound diagnosis device 300 may transmit the ultrasoundimage to the mobile terminal through a short-range wirelesscommunication network using a communication module included in theultrasound diagnosis device 300, so that the original fetal image may bedisplayed and stored in real-time.

The VR device 400 may be wiredly or wirelessly connected to the imageproviding server 200 to reproduce the 3D fetal image provided by themobile terminal 100. As the VR device 400, a head-mounted deviceequipped with, e.g., a geomagnetic sensor or an acceleration sensor, aswell as a display, may be used, and may implement a stereoscopic imagefor the fetus corresponding to the direction in which the user views thediagnosis target.

Further, the UR device 400 may be configured as a housing member inwhich a predetermined reproducing terminal capable of reproducing thevirtual reality image is installed, and be implemented with a structureof having the mobile terminal 100 mounted thereon to produce the 3Dfetal image.

According to the above-described structure, the deep learning-basedvirtual reality 3D fetal model providing system according to anembodiment of the present invention may provide the original fetal imageacquired through the ultrasound diagnosis device to the image providingserver through the information communication network, generate a 3Dfetal model according to a modeling process to which a deep learningscheme is applied, and display the 3D fetal model through the VR device,thereby providing the 3D fetal model which is closer to the actualappearance.

Hereinafter, the mobile terminal of the deep learning-based virtualreality 3D fetal model providing system is described with reference tothe drawings.

FIG. 2 is a view illustrating a structure of a mobile terminal of a deeplearning-based virtual reality 3D fetal model providing system accordingto an embodiment of the present invention. In the following description,each component constituting the mobile terminal may be implemented as anapplication program executable by a predetermined microprocessor andrecorded in a recording medium.

Referring to FIG. 2, the mobile terminal 100 of the present inventionmay include a communication unit 110 connected to an informationcommunication network to transmit/receive data with an external system,an image storage device 120 storing fetal images transmitted from anultrasound diagnosis device through the communication unit 110, an ROIsetting unit 130 receiving a selection of a facial area in the fetalimage according to the user's input and setting the same as ROIinformation, a growth information input unit 140 receiving growthinformation including the pregnancy term and growth parameters of thepregnant woman, a terminal library DB 150 storing a library in whichcomponents corresponding to a plurality of facial shapes are defined,and a feedback unit 160 receiving one or more components for each facialportion of the fetus from the user and transmitting a codeword of asimilar model to the image providing server.

The communication unit 110 may connect to the information communicationnetwork to transmit/receive data to and from an external system. Thecommunication unit 110 may receive an original fetal image from theultrasound diagnosis device, provide an original fetal image for whichan ROI has been set to the image providing server, or receive a 3D fetalimage from the image providing server.

The image storage device 120 may store original fetal images transmittedfrom the ultrasound diagnosis device and 3D fetal images provided fromthe image providing server. In particular, as fetal images areclassified and stored by date, the user may easily identify changes inthe fetus from the past to the present. The 3D fetal image stored in theimage storage device may be reproduced on the mobile terminal 100 ortransmitted to an external VR device to be reproduced in the form of astereoscopic image.

The ROI setting unit 130 may provide an interface through which the usermay set a region of interest (ROI) for the fetal image through regionselection. The user, e.g., the pregnant woman or her guardian, maydesignate an ROI through such an action as dragging the areacorresponding to the fetus's face in the original fetal image displayedthrough the screen of the mobile terminal 100, and the ROI setting unit130 may generate ROI information for the corresponding fetal image.

The growth information input unit 140 may receive growth information forthe ROI-set fetal image. In addition to uploading the fetal image andROI information, the user may input, as the growth information, the timeof photographing the currently uploaded fetal image, that is, thepregnancy term, and various information related to the growth of thefetus.

The fetal image, ROI information, and growth information may besynchronized and transmitted to the image providing system through thecommunication unit 110. Accordingly, the image providing system performsa 3D modeling process using the gathered data and generates a 3D fetalmodel of the fetus.

The terminal library DB 150 may store various types of 3D componentscorresponding to each part of the fetus's face.

The deep learning-based virtual reality 3D fetal model providing systemaccording to an embodiment of the present invention is characterized byenhancing the performance of the deep learning model through userfeedback. Accordingly, the user terminal 100 stores a number of 3Dcomponents, which are various types of models for each feature part,i.e., eyes, nose, mouth, and face, of the fetus's face, and the user mayselect the component most similar to the shape of her fetus from amongthe 3D components stored in the terminal library DB 150 and generate asimilar face model and provide an image and codeword for the similarface model to the image providing server.

Accordingly, the image providing server may use the codeword of thefetus's similar face model provided from the user terminal 100, astraining data of the deep learning model, thereby enhancing the accuracyof 3D modeling on the fetus.

The feedback unit 160 may extract 3D components for each facial portionstored in the terminal library DB 150 and provide an interface throughwhich the user may make a designation. Further, when the components forthe feature parts of the fetus's face are selected by the user, thefeedback unit 160 generates a similar face model by combining theselected components and provide the codeword of the similar face modelto the image providing server through the communication unit 110.

By the above-described structure, the mobile terminal according to theembodiment of the present invention may provide the fetal image to theimage providing server and make a request for a 3D fetal model mostsimilar to the actual fetus to the image providing server, and themobile terminal may also feed back information on the similar facemodel, thereby maximizing the similarity in 3D modeling.

Hereinafter, the image providing server of the deep learning-basedvirtual reality 3D fetal model providing system is described withreference to the drawings.

FIG. 3 is a view illustrating a structure of an image providing serverof a deep learning-based virtual reality 3D fetal model providing systemaccording to an embodiment of the present invention, and FIG. 4 is aview illustrating a process for increasing the matching accuracy of a 3Dmodel by reflecting user feedback in a deep learning-based virtualreality 3D fetal model providing system according to an embodiment ofthe present invention.

Referring to FIGS. 3 and 4, the image providing server 200 according toan embodiment of the present invention may include an image receivingunit 210 receiving a fetal image and ROI information from the mobileterminal 100, an image processor 220 inputting the fetal imagecorresponding to the ROI information to a deep learning model to therebyextract features per facial portion, a database 230 storing a serverlibrary to which the deep learning model refers, a 3D model generator240 generating a 3D fetal model reflecting the per-facial portionfeatures through 3D modeling, and a member management unit 250determining whether a user is registered as a member according to logininformation entered from the mobile terminal 100 and performs a loginprocess.

The image receiving unit 210 may receive original fetal images from oneor more mobile terminals 100 through an information communicationnetwork. The image receiving unit 210 may interwork with an applicationprogram installed on the mobile terminal 100 to receive a 3Dstereoscopic image for the fetus or to periodically receive fetal imagesfrom the user who intends to manage the fetal images according to thegrowth period of the fetus.

Here, the image receiving unit 210 receives the original fetal image,ROI information set in the original fetal image, and information on thecurrent fetal growth period together, thereby enabling the appearance ofthe fetus for each growth period to be stored and managed.

The image processor 220 may generate a 3D fetal model for virtualreality by performing 3D modeling using the original fetal imagereceived by the image receiving unit 210. The image processor 220 mayextract facial portions of the fetus by referring to the ROI informationin the original fetal image, and extract per-facial portion componentsmost similar to each part of the face for 3D modeling.

The fetus's face in the 3D ultrasound fetal image look unclear for theeyes, nose, and mouth, and the facial portions may be blocked byfloating materials in the womb or according to the fetus's position.Thus, a training model optimized for the fetal image is required, andthe image processor 220 extracts per-facial portion components whichexhibit the minimum error or difference from the actual face through aknown deep learning model.

Further, the image processor 220 may increase the accuracy of matchingof the 3D model by updating the deep learning model using the codewordof the similar face model, as feedback information transmitted from themobile terminal 100, as training data for the deep learning model.

The database 230 may store various types of information for 3D modeling.In particular, the image processor 220 refers to the library for 3Dmodeling and requires a plurality of training data for the deep learningmodel. The database 230 may store various types of information necessaryfor such 3D modeling.

This database 230 may include a plurality of database systems logicallyor physically divided according to their purposes, and the database 230may include a member DB 231 for storing member and fetal information andmanaging fetal images according to the growth period of the fetus 231, aserver library DB 232 storing a library including components for eachfacial portion of the fetus, and a deep learning DB 233 storing trainingdata used for machine learning.

The model generator 240 may update the face of a standard 3D fetal modelby combining the plurality of per-facial portion components derived bythe image processor 220.

The member management unit 250 may provide functions, such as amembership registration process, login process, and member informationmanagement of pregnant women or guardians who wish to use the 3D fetalimages provided by the system of the present invention.

In particular, the user may access the image providing server 200 fromtime to time to manage and identify the image of the fetus for eachpregnancy period stored in the storage allocated to his account, and themember management unit 250 may perform a login procedure according to arequest from the mobile terminal 100 and identify the 3D fetal image forthe user, stored in the storage of the database 230.

According to the above-described structure, the image providing serverof the deep learning-based virtual reality 3D fetal model providingsystem according to an embodiment of the present invention may provide a3D model for the fetal image provided by the user and continuouslyenhance the accuracy of machine learning through feedback from the user,thereby providing a more realistic fetal model.

Hereinafter, the image processor included in the image providing serveraccording to an embodiment of the present invention is described withreference to the drawings.

FIG. 5 is a view illustrating a structure of an image processor of animage providing server according to an embodiment of the presentinvention, and FIG. 6 is a view illustrating a 3D modeled image of animage providing server according to an embodiment of the presentinvention.

Referring to FIGS. 5 and 6, the image processor 220 according to anembodiment of the present invention may include a preprocessor 221designating one or more frames in a received fetal image and removingnoise, an ROI matching unit 222 determining a facial area which is atarget for modeling by matching ROI information onto the frame, a deeplearning model 224 for performing a machine learning process on the setdeep learning model using the codewords corresponding to the pluralityof components, as training data, or extracting facial feature codewordsfor the input image after training, and a component selector 225exchanging one or more components per facial portion included in theexisting 3D model, corresponding to the codeword.

The preprocessor 221 may designate a frame of a predetermined range tobe subjected to 3D modeling according to settings, for the input fetalimage, and may remove noise for the designated frame.

The ROI matching unit 222 may determine the facial portion by matchingROI information designated by the user onto the noise-removed frame. TheROI information may include coordinates for the facial area of the fetusselected by the user in the fetal image, and the ROI matching unit 222may extract the image corresponding to the facial area through the ROIinformation.

The above-described components may be defined by classifying the shapeof each facial portion in a predetermined number, and the components maybe classified into three types for each facial portion as illustrated inTable 1 below.

TABLE 1 Feature part Classification criteria Type 1 Type 2 Type 3 Eyeshape Degree of protrusion sunken normal protruding of eye area Noseshape how high and broad low and broad normal high Mouth degree ofprotrusion upper normal lower shape of upper/lower lips protrusionprotrusion Face shape degree of slimness slim normal round of cheeks andchin

Here, the shape of each facial portion may correspond to any one of thethree types, and have a binary-format value designated thereto. As anexample, if the eye shape is normal, then the codeword becomes ‘010’,and if the face shape is slim, the codeword becomes ‘100’. The deeplearning model 224 may receive the extracted codewords as training dataand perform machine learning, and the per-facial portion featureinformation corresponding to the facial area of the fetal image by theROI matching unit 222 is labeled, and training data is generated.

Specifically, the deep learning model 224 may operate in a learning modeor a feature extraction mode. When operating in the training mode, thedeep learning model 224 uses a number of images input by the user andsimilar model codewords matching the images, for training. Here, thecodeword of the similar model may be used only when learning based onuser feedback.

Further, when operating in the feature extraction mode, the deeplearning model 224 outputs the feature as the codeword based on thelearned information if a fetal image is input.

To that end, image frames for training may be designated in the fetalimage through the preprocessor 211 and the ROI matching unit 222, andROI information for designating the fetus's facial area in the imagearea of each frame may be set, so that the facial area of the fetus inthe ROI may be utilized as an input image for training. Since the 3Dfetal image typically includes the fetus's movement, there is anadvantage that a large number of images for training may be obtainedfrom one video.

Here, to designate the feature of the fetus's facial portion, a featuredesignation function using a similar 3D model may be implemented. Theface shape model most similar to the face shape, eyes, nose, and mouthin the facial area of the fetal image and a model for the facial portionmay be selected from the library, so that a 3D face model overallclosest to the face in the fetal image may be determined. To that end,various facial shapes and models for the facial portions are produced ascomponents, and model selection and replacement, rotation or movementmay be carried out for easy comparison with the fetal image.

Here, the generated images for training and the feature codewords forthe fetus's face may be constructed as a learning database forintegrated management thereof so as to facilitate addition ormodification to the information later.

The component selector 225 may replace one or more components for eachfacial portion included in the existing 3D model, corresponding to thecodeword, so that 3D modeling may be performed using the updatedcomponents upon generating a 3D fetal model.

According to the above-described structure, the image processoraccording to an embodiment of the present invention may providecomponents for generating a 3D fetal model, and may reflect feedback fora similar model provided from the user to the deep learning model 224.In this case, the component may be delivered in the form of a codeword.

Hereinafter, a deep learning-based virtual reality 3D fetal modelproviding method according to an embodiment of the present invention isdescribed with reference to the drawings.

FIG. 7 is a view illustrating a method of providing a virtual reality 3Dfetal model based on deep learning according to an embodiment of thepresent invention. Unless specifically stated, each step described belowis performed by the above-described image providing server or itscomponents.

Referring to FIG. 7, a method for providing a virtual reality 3D fetalmodel based on deep learning according to an embodiment of the presentinvention may include the step S100 of receiving a fetal image and ROIinformation from a mobile terminal, the step S110 of extractingper-facial portion features by inputting a fetal image corresponding tothe ROI information to a deep learning model, the step S120 ofgenerating a 3D fetal image reflecting the per-facial portion featuresthrough 3D modeling, and the step S130 of providing the 3D fetal imageto the mobile terminal.

In the step S100 of receiving the fetal image and ROI information fromthe mobile terminal, the mobile terminal receives and stores an originalfetal image of the fetus from the ultrasound diagnosis device, generatesROI information according to the user's designation, and transmits theROI information along with the stored original fetal image to the imageproviding server.

In the step of extracting per-facial portion features by inputting thefetal image corresponding to the ROI information to the deep learningmodel, the image providing server extracts the facial area in the fetalimage according to the image coordinates included in the ROIinformation, and extracts features for each portion in the facial areathrough the deep learning model.

In the step S120 of generating a 3D fetal model reflecting theper-facial portion features through 3D modeling, the deep learning modelfeatures the per-facial portion features and generates one 3D fetalmodel.

In the step S130 of providing the 3D fetal model to the mobile terminal,the image providing server provides the generated 3D fetal model to themobile terminal.

Accordingly, the user may enjoy a stereoscopic fetal image at variousviewpoints for the fetus through the mobile terminal, and may also enjoya 3D fetal model in a virtual reality manner through a VR deviceinterworking with the mobile terminal.

As steps for the user's feedback to enhance the performance of the deeplearning model after step S130, the method may further include the stepof receiving a codeword for a similar model including one or morecomponents for each facial portion of the fetus elected by the user fromthe mobile terminal and the step of performing a machine learningprocess on the deep learning model using the codeword as training data,after providing the 3D fetal image to the mobile terminal.

Hereinafter, the technical spirit of the present invention is describedthrough a deep learning process by a deep learning-based virtual realityfetal image providing system according to an embodiment of the presentinvention and a process of extracting a codeword in the process.

FIG. 8 is a view schematically illustrating a deep learning and codewordextraction process of a deep learning-based virtual reality fetal imageproviding system according to an embodiment of the present invention.

Referring to FIG. 8, the deep learning-based virtual reality fetal imageproviding system according to an embodiment of the present invention isconfigured to use various fetal images as training data, prepare animage for each fetus, designate ROI information corresponding to theface in the image, and apply the same to the deep learning model tothereby perform 3D modeling.

To train the deep learning model, a number of training images andcodewords reflecting the facial features of the images are previouslydefined, and the deep learning model training process is performed. Thetraining process is a process of optimizing the deep learning model toreduce errors so that a predefined image and a codeword matching theretoare output (a).

Here, as binarized data, identification and extraction of a componentmay be performed via a codeword corresponding thereto.

Accordingly, if the user provides the current fetal image and ROIinformation, the codeword of the component corresponding to each facialportion is extracted and a 3D model is generated through the learneddeep learning model reflecting the user's feedback (b).

According to an embodiment of the present invention, it is possible toreduce errors and enhance the quality of the 3D model by updatingtraining data of the deep learning model according to the user'sfeedback.

To that end, if a library including a plurality of components isprovided to the user's mobile terminal, and a similar model is generatedby the user's selection of the most similar components from theper-facial portion components provided from the library, and a codewordis provided, the image providing server uses the same as training data.

While the disclosure has been shown and described with reference toexemplary embodiments thereof, it will be apparent to those of ordinaryskill in the art that various changes in form and detail may be madethereto without departing from the spirit and scope of the disclosure asdefined by the following claims. Accordingly, the scope of thedisclosure should be defined by the following claims and equivalentsthereof, but not by the above-described embodiments.

LEGEND OF REFERENCE NUMBERS

100: mobile terminal 110: communication unit 120: image storage device130: ROI setting unit 140: growth information input unit 150: terminallibrary DB 160: feedback unit 200: image providing server 210: imagereceiving unit 220: image processor 221: preprocessor 222: ROI matchingunit 224: deep learning model 225: component selector 230: database 240:3D model generator 250: member management unit 300: ultrasound diagnosisdevice 400: UR device

1. A system for providing a virtual reality 3D fetal model based on deeplearning, the system comprising an image providing server generating a3D model by analyzing a fetal image provided from a mobile terminal, theimage providing server including: an image receiving unit receiving afetal image and ROI information from the mobile terminal; an imageprocessor inputting a fetal image corresponding to the ROI informationto a deep learning model and extracting per-facial portion features; anda 3D model generator generating a 3D fetal model reflecting theper-facial portion features by selecting and synthesizing part modelsaccording to the per-facial portion features.
 2. The system of claim 1,wherein the image processor includes: a preprocessor designating one ormore frames from the input fetal image and removing noise; and an ROImatching unit determining a facial area, as a target for modeling, bymatching the ROI information onto the frame.
 3. The system of claim 2,wherein the image receiving unit receives a codeword for a similar modelcomposed of a plurality of components selected from the mobile terminalalong with reception of the fetal image and the ROI information, andwherein the image processor includes a deep learning model forperforming a machine learning process using the codeword correspondingto the plurality of components, as training data, or extracting a facialfeature codeword for an input image after training.
 4. The system ofclaim 3, wherein the facial area includes one or more of an eye, a nose,a mouth, and a face, and wherein the codeword is a code resultant from ashape of the facial area in a binary format.
 5. The system of claim 4,wherein the image processor includes a component selector exchanging oneor more components for each facial area included in an existing 3Dmodel, corresponding to the codeword.
 6. The system of claim 1, whereinthe image providing server includes a member management unit determiningwhether a user is registered as a member according to login informationinput from the mobile terminal and performing a login process, andwherein the image processor stores the fetal image in storage allocatedto a user identified by the member management unit and extracts thestored 3D fetal image.
 7. The system of claim 1, wherein the imageprocessor receives growth information including a pregnancy period andgrowth parameter for a pregnant woman from the mobile terminal, andwherein the 3D model generator stores the generated 3D model in adatabase according to the growth parameter per pregnancy period.
 8. Thesystem of claim 1, wherein the mobile terminal includes: a communicationunit connected to an information communication network to communicatewith an external system; a storage device storing the fetal imagetransmitted from an ultrasound diagnosis device through thecommunication unit; an ROI setting unit receiving a selection of afacial area in the fetal image according to a user input and setting thefacial area as the ROI information; a growth information input unitreceiving growth information including a pregnancy period and growthparameter for a pregnant woman; a terminal library DB storing a libraryin which components corresponding to a plurality of face shapes aredefined; and a feedback unit receiving a selection of one or morecomponents for facial portion of the fetus from a user and transmittinga codeword of a similar model to the image providing server.
 9. A methodfor providing a model by an image providing server of a deeplearning-based virtual reality 3D fetal model providing system accordingto claim 1, the method comprising: receiving a fetal image and ROIinformation from a mobile terminal; extracting per-facial portionfeatures by inputting a fetal image corresponding to the ROI informationto a deep learning model; generating a 3D fetal image reflecting theper-facial portion features through 3D modeling; and providing the 3Dfetal image to the mobile terminal.
 10. The method of claim 9, furthercomprising: after providing the 3D fetal image to the mobile terminal,receiving a codeword for a similar model including one or morecomponents per facial portion of the fetus selected by a user from themobile terminal; and performing a machine learning process on the deeplearning model using the codeword as training data.